Add docs for UI Agents#772
Open
markbackman wants to merge 4 commits into
Open
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Updates the docs to align with the UI Agent Protocol shipped across pipecat#4407, pipecat-client-web#203, and pipecat-ai-subagents#18. The wire format moved into pipecat as canonical (5 first-class RTVI message types, paired Data/Message envelopes, 5 new pipeline frames, PROTOCOL_VERSION 1.2.0 → 1.3.0); subagents builds the agent abstractions on top; client-web ships UIAgentClient + React idioms. Pipecat (server) - client/rtvi-standard.mdx: bumped version note to 1.3, added a UI Agent Protocol section documenting all 5 message types (ui-event, ui-snapshot, ui-cancel-task, ui-command, ui-task) with shapes and the 8 standard payload models. - api-reference/server/rtvi/rtvi-processor.mdx: added a UI Agent Protocol section documenting the on_ui_message handler and the inbound / outbound frame conventions. - api-reference/server/rtvi/rtvi-observer.mdx: extended the frame-translation table for RTVIUICommandFrame and RTVIUITaskFrame. - api-reference/server/frames/system-frames.mdx: added a RTVI UI Frames section documenting all 5 new pipeline frames. Subagents - api-reference/pipecat-subagents/ui-agent.mdx: fixed parent class (LLMContextAgent), refreshed bridge frame references to RTVIUICommandFrame / RTVIUITaskFrame, added keep_history param, added 5 action helpers (scroll_to / highlight / select_text / click / set_input_value), added user_task_group / start_user_task_group / reset_context methods, added on_task_cancelled lifecycle hook, documented single-flight task lock semantics. - api-reference/pipecat-subagents/ui-tools.mdx: full rewrite for ReplyToolMixin (the prior ScrollToToolMixin / HighlightToolMixin documentation no longer matched the source). - api-reference/pipecat-subagents/ui-commands.mdx: fixed import path to point at pipecat (canonical home), added Click, SetInputValue, SelectText. - api-reference/pipecat-subagents/ui-prompts.mdx: fixed import path, added <selection> block bullet. - api-reference/pipecat-subagents/messages.mdx: added 4 task-lifecycle bus messages (BusUITaskGroupStartedMessage, BusUITaskUpdateMessage, BusUITaskCompletedMessage, BusUITaskGroupCompletedMessage), refreshed attach_ui_bridge to reflect the new wiring (subscribes to on_ui_message, emits RTVIUICommandFrame / RTVIUITaskFrame), removed stale UI_*_MESSAGE_TYPE constants section. - api-reference/pipecat-subagents/overview.mdx: added LLMContextAgent row to the agent hierarchy and fixed UIAgent's parent. - subagents/learn/ui-agent.mdx: NEW consolidated guide. Sections: Why, Architecture, How information flows, Sequence diagram, Action vocabulary, Choosing a tool shape, Async tasks and lifecycle, When to use, Reference app, API reference. Sourced from the internal design doc and the music-player demo. - subagents/examples/overview.mdx: added a Reference Applications section linking pipecat-music-player. - docs.json: registered subagents/learn/ui-agent in the navigation. Client (js + react) - api-reference/client/js/ui-agent-client.mdx: refreshed to RTVI v1.3 (RTVIEvent.UICommand instead of RTVIEvent.ServerMessage, kebab-case wire types), added addTaskListener / removeTaskListener / removeAllTaskListeners / cancelTask, added See-also pointers. - api-reference/client/js/a11y-snapshot.mdx: replaced the dropped UI_SNAPSHOT_EVENT_NAME section with A11ySelection shape docs and a See-also section noting the streamer dispatches via RTVIMessageType.UI_SNAPSHOT. - api-reference/client/js/client-methods.mdx: added sendRTVIMessage section under Messages. - api-reference/client/react/hooks.mdx: added useStandardSelectTextHandler, useStandardSetInputValueHandler, useStandardClickHandler, useUITasks; updated useStandardCommandHandlers to reflect that it installs all six handlers. - api-reference/client/react/components.mdx: added UITasksProvider; refreshed UIAgentProvider description for RTVIEvent.UICommand / RTVIEvent.UITask. Verified with mint broken-links (no broken links found) and prettier --write across all touched files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation update for the UI Agent Protocol that shipped across these four PRs:
on_ui_messagehandler, payload models,PROTOCOL_VERSION1.2.0 → 1.3.0).UIAgentClientplus React idioms (UIAgentProvider,UITasksProvider, hooks, standard handlers,A11ySnapshotStreamer).UIAgent(now extendsLLMContextAgent), expanded action helpers,ReplyToolMixin,start_user_task_groupfor streaming worker fan-out,keep_historyflag,attach_ui_bridgerewired toon_ui_message.The wire format moved into Pipecat as the canonical authority, so single-LLM apps can target it directly without taking a subagents dependency. Subagents builds the agent abstractions on top.
Changes
Pipecat (server)
client/rtvi-standard.mdx— bumped version note to 1.3, added a UI Agent Protocol section documenting all 5 message types (ui-event,ui-snapshot,ui-cancel-task,ui-command,ui-task) with shapes, plus a Standard Command Payloads table for the 8 standard payload models.api-reference/server/rtvi/rtvi-processor.mdx— added a UI Agent Protocol section documenting the newon_ui_messageevent handler and the inbound / outbound frame conventions.api-reference/server/rtvi/rtvi-observer.mdx— extended the frame-translation table forRTVIUICommandFrameandRTVIUITaskFrame.api-reference/server/frames/system-frames.mdx— added a RTVI UI Frames section documenting all 5 new pipeline frames (RTVIUICommandFrame,RTVIUITaskFrame,RTVIUIEventFrame,RTVIUISnapshotFrame,RTVIUICancelTaskFrame).Subagents
api-reference/pipecat-subagents/ui-agent.mdx— fixed parent class toLLMContextAgent, refreshed bridge frame references, addedkeep_history, added the 5 action helpers (scroll_to,highlight,select_text,click,set_input_value), addeduser_task_group/start_user_task_group/reset_context, addedon_task_cancelledlifecycle hook, documented single-flight task lock semantics.api-reference/pipecat-subagents/ui-tools.mdx— full rewrite forReplyToolMixin(the priorScrollToToolMixin/HighlightToolMixindocumentation no longer matched the source).api-reference/pipecat-subagents/ui-commands.mdx— fixed import path to point atpipecat.processors.frameworks.rtvi.models(canonical home), addedClick,SetInputValue,SelectText.api-reference/pipecat-subagents/ui-prompts.mdx— fixed import path, added<selection>block bullet.api-reference/pipecat-subagents/messages.mdx— added 4 task-lifecycle bus messages (BusUITaskGroupStartedMessage,BusUITaskUpdateMessage,BusUITaskCompletedMessage,BusUITaskGroupCompletedMessage), refreshedattach_ui_bridgeto reflect the new wiring (subscribes toon_ui_message, emitsRTVIUICommandFrame/RTVIUITaskFrame), removed staleUI_*_MESSAGE_TYPEconstants section.api-reference/pipecat-subagents/overview.mdx— addedLLMContextAgentrow to the agent hierarchy and fixedUIAgent's parent.subagents/learn/ui-agent.mdx— consolidated UI Agent guide. Sections: Why, Architecture (voice + UI agent split), How information flows (4 loops), Sequence diagram (voice + UI for action), Action vocabulary (pointing / reading / writing / presentation), Choosing a tool shape (ReplyToolMixinvs custom@tool replyvs per-domain tools), Async tasks and lifecycle (start_user_task_group+useUITasks+ cancel), When to use (deployment shapes), Reference app, API reference. Sourced from the internal design doc and the music-player demo.subagents/examples/overview.mdx— added a Reference Applications section linkingpipecat-music-player.docs.json— registeredsubagents/learn/ui-agentin the navigation.Client (js + react)
api-reference/client/js/ui-agent-client.mdx— refreshed to RTVI v1.3 (RTVIEvent.UICommandinstead ofRTVIEvent.ServerMessage, kebab-case wire types), addedaddTaskListener/removeTaskListener/removeAllTaskListeners/cancelTask.api-reference/client/js/a11y-snapshot.mdx— replaced the droppedUI_SNAPSHOT_EVENT_NAMEsection withA11ySelectionshape docs and a See-also section noting the streamer dispatches viaRTVIMessageType.UI_SNAPSHOT.api-reference/client/js/client-methods.mdx— addedsendRTVIMessage(msgType, data?)section under Messages.api-reference/client/react/hooks.mdx— addeduseStandardSelectTextHandler,useStandardSetInputValueHandler,useStandardClickHandler,useUITasks; updateduseStandardCommandHandlersto reflect that it installs all six handlers.api-reference/client/react/components.mdx— addedUITasksProvider; refreshedUIAgentProviderdescription forRTVIEvent.UICommand/RTVIEvent.UITask.Verification
mint broken-links— no broken links found across the entire docs site.npx prettier --write— applied to all touched files.